Transformer2026-10-12 04:20:34BUPT team’s MiTA reframes efficient attention through a fast-weight FFN lensA research team from Beijing University of Posts and Telecommunications has introduced Mixture-of-Top-k Attention, or MiTA, a method that recasts standard attention as a feed-forward network whose weights are generated by the current input and whose hidden width grows with sequence length. The paper, accepted to NeurIPS 2026, uses that fast-weight view to build a five-dimensional taxonomy for efficient attention and then proposes a design that combines reusable routed experts with a shared global expert. In the paper’s formulation, efficient attention methods fall into two broad paths: compression, which summarizes a growing fast-weight network into compact modules, and routing, which preserves raw content but only visits selected parts. MiTA tries to bridge the two. It extracts a small set of landmark queries, lets them build Top-k routed experts from the full context, and also uses the same landmarks to construct a shared global representation. Each query then reuses an expert rather than assembling its own private Top-k set. The reported results cover ImageNet-1K classification, ADE20K segmentation, Long Range Arena, and arXiv long-document classification. The paper says MiTA reaches 79.8% Top-1 on DeiT-S versus 79.9% for full attention while cutting FLOPs from 4.6G to 4.4G, reduces total LRA training time from 10.6 hours to 2.4 hours, and posts 88.0%, 92.9%, and 98.7% accuracy on arXiv classification at 16K, 32K, and 64K lengths.290
Pluralis2026-09-25 19:53:07Pluralis team has three papers accepted to NeurIPS 2026Techub News reported that @coinfund said in a post on X that the team at AI research company Pluralis had three papers accepted to NeurIPS 2026. CoinFund also congratulated Pluralis co-founder Alexander Long and the broader team. NeurIPS is widely regarded as a top academic conference in artificial intelligence and machine learning. The update did not disclose the paper titles or research topics, and the announcement was limited to the acceptance result and the congratulatory message shared by @coinfund.360
